feat: support json index #36750

sunby · 2024-10-10T10:46:53Z

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later.

basic usage:

collection.create_index("json_field", {"index_type": "INVERTED",
    "params": {"json_cast_type": DataType.STRING, "json_path":
'json_field["a"]["b"]'}})

There are some limits to use this index:

If a record does not have the json path you specify, it will be ignored and there will not be an error.
If a value of the json path fails to be cast to the type you specify, it will be ignored and there will not be an error.
A specific json path can have only one json index.
If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version.

sre-ci-robot · 2024-10-10T10:47:09Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sunby
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-10-10T10:48:06Z

@sunby Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

mergify · 2024-10-10T11:11:52Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-10T11:17:16Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-10-12T04:00:08Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

tedxu · 2024-10-14T07:34:35Z

internal/core/src/segcore/SegmentInterface.h

@@ -201,6 +202,16 @@ class SegmentInternalInterface : public SegmentInterface {
        return *ptr;
    }

+    template <typename T>
+    const index::ScalarIndex<T>&
+    chunk_scalar_index(std::string path, int64_t chunk_id) const {


The path argument passed in as nested_path may be duplicate between 2 different fields, we should introduce an additional fieldId argument here.

tedxu · 2024-10-14T10:13:01Z

internal/datacoord/index_service.go

+	}
+
+	// check duplicate json path
+	exists := s.meta.indexMeta.GetFieldIndexes(req.GetCollectionID(), req.GetFieldID(), req.GetIndexName())


Is it allowed to create duplicated indexes with the same json path and different index names?

tedxu · 2024-10-14T10:46:24Z

internal/datacoord/index_service.go

+	return nil
+}
+
+func (s *Server) parseNestedPath(identifier string, schema *schemapb.CollectionSchema) (string, error) {


Can we document the format of identifier?

PS, the code would be much easier to read if the parse is replaced with regex match.

Can we document the format of identifier?

PS, the code would be much easier to read if the parse is replaced with regex match.

it's copied from parser implentation

tedxu · 2024-10-14T10:53:15Z

internal/querynodev2/segments/segment_loader.go

-			if err != nil {
-				log.Warn("failed to load index for segment", zap.Error(err))
-				return err
+			if info.IsJson {


Do we need to change the getResourceUsageEstimateOfSegment function per quantity and size of json indexes?

Do we need to change the getResourceUsageEstimateOfSegment function per quantity and size of json indexes?

we use inverted index to implement json indexes so we can reuse the memroy estimation of inverted index.

mergify · 2024-10-31T09:41:37Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-10-31T10:36:01Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2024-10-31T10:54:12Z

Codecov Report

Attention: Patch coverage is 53.30396% with 106 lines in your changes missing coverage. Please review.

Project coverage is 67.19%. Comparing base (7cfd609) to head (43e288f).
Report is 12 commits behind head on master.

❗ Current head 43e288f differs from pull request most recent head b6add46

Please upload reports for the commit b6add46 to get more accurate results.

Files with missing lines	Patch %	Lines
internal/core/src/index/IndexFactory.cpp	0.00%	34 Missing ⚠️
...core/src/indexbuilder/JsonInvertedIndexCreator.cpp	0.00%	21 Missing ⚠️
internal/core/src/exec/expression/UnaryExpr.cpp	59.52%	17 Missing ⚠️
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp	0.00%	11 Missing ⚠️
internal/core/src/index/InvertedIndexTantivy.h	76.47%	4 Missing ⚠️
internal/core/src/segcore/SegmentInterface.h	55.55%	4 Missing ⚠️
internal/core/src/segcore/load_index_c.cpp	0.00%	4 Missing ⚠️
internal/core/src/index/InvertedIndexTantivy.cpp	0.00%	3 Missing ⚠️
...l/core/src/indexbuilder/JsonInvertedIndexCreator.h	84.21%	3 Missing ⚠️
...ernal/core/src/indexbuilder/ScalarIndexCreator.cpp	25.00%	3 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #36750       +/-   ##
===========================================
- Coverage   83.23%   67.19%   -16.05%     
===========================================
  Files        1015      292      -723     
  Lines      157697    25578   -132119     
===========================================
- Hits       131260    17186   -114074     
+ Misses      21228     8392    -12836     
+ Partials     5209        0     -5209

Components	Coverage Δ
Client	`∅ <ø> (∅)`
Core	`67.19% <52.04%> (∅)`
Go	`∅ <ø> (∅)`

Files with missing lines	Coverage Δ
internal/core/src/common/FieldDataInterface.h	`56.76% <100.00%> (ø)`
internal/core/src/common/FieldMeta.h	`94.64% <ø> (ø)`
...e/src/exec/expression/BinaryArithOpEvalRangeExpr.h	`100.00% <100.00%> (ø)`
...nternal/core/src/exec/expression/BinaryRangeExpr.h	`92.68% <100.00%> (ø)`
internal/core/src/exec/expression/ExistsExpr.h	`100.00% <100.00%> (ø)`
internal/core/src/exec/expression/Expr.h	`69.07% <100.00%> (ø)`
...ternal/core/src/exec/expression/JsonContainsExpr.h	`100.00% <100.00%> (ø)`
internal/core/src/exec/expression/TermExpr.h	`85.71% <100.00%> (ø)`
internal/core/src/exec/expression/UnaryExpr.h	`77.77% <100.00%> (ø)`
internal/core/src/index/IndexFactory.h	`100.00% <ø> (ø)`
... and 14 more

... and 1283 files with indirect coverage changes

mergify · 2024-10-31T10:59:20Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

xiaofan-luan · 2024-11-01T00:24:03Z

@sunby

Are you done with this feature so I can start to review it?

mergify · 2024-11-04T07:07:56Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-04T07:07:56Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-04T07:14:05Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-05T09:00:37Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-05T09:01:18Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-05T09:04:54Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-06T04:03:38Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-06T04:33:37Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-06T05:08:02Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-06T07:25:23Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-06T07:31:04Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-06T07:31:04Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-06T09:15:17Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-06T09:53:17Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-06T11:51:26Z

@sunby go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-06T13:28:24Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-07T05:42:33Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

This PR adds json index support for json and dynamic fields. Now you can only do unary query like 'a["b"] > 1' using this index. We will support more filter type later. basic usage: ``` collection.create_index("json_field", {"index_type": "INVERTED", "params": {"json_cast_type": DataType.STRING, "json_path": 'json_field["a"]["b"]'}}) ``` There are some limits to use this index: 1. If a record does not have the json path you specify, it will be ignored and there will not be an error. 2. If a value of the json path fails to be cast to the type you specify, it will be ignored and there will not be an error. 3. A specific json path can have only one json index. 4. If you try to create more than one json indexes for one json field, sdk(pymilvus<=2.4.7) may return immediately because of internal implementation. This will be fixed in a later version. Signed-off-by: sunby <[email protected]>

Signed-off-by: sunby <[email protected]>

mergify · 2024-11-07T06:58:57Z

@sunby cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-07T07:14:03Z

@sunby E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sre-ci-robot added the area/internal-api label Oct 10, 2024

sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Oct 10, 2024

sre-ci-robot requested review from aoiasd and bigsheeper October 10, 2024 10:47

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Oct 10, 2024

sunby force-pushed the add_json_index branch from 0f2e61d to e2a08b7 Compare October 10, 2024 10:47

mergify bot added the do-not-merge/missing-related-issue label Oct 10, 2024

sunby force-pushed the add_json_index branch 2 times, most recently from 7ececff to 68a0644 Compare October 10, 2024 10:52

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Oct 12, 2024

tedxu reviewed Oct 14, 2024

View reviewed changes

sunby force-pushed the add_json_index branch from ff47ada to 7eac44a Compare October 31, 2024 09:07

sunby force-pushed the add_json_index branch from bf1de33 to f87196f Compare November 5, 2024 08:16

sre-ci-robot added the area/compilation label Nov 5, 2024

sre-ci-robot added area/test sig/testing labels Nov 6, 2024

sunby force-pushed the add_json_index branch from 0e62cca to 43e288f Compare November 7, 2024 04:15

sunby added 10 commits November 7, 2024 14:45

fix

a4360b2

Signed-off-by: sunby <[email protected]>

fix

1ba0549

Signed-off-by: sunby <[email protected]>

fix

805df78

Signed-off-by: sunby <[email protected]>

fix

d2aea02

Signed-off-by: sunby <[email protected]>

fix code checker

0ffb51c

Signed-off-by: sunby <[email protected]>

fix

ac0d4ce

Signed-off-by: sunby <[email protected]>

fix test

0cadfb0

Signed-off-by: sunby <[email protected]>

skip some go client tests

a857eda

Signed-off-by: sunby <[email protected]>

fix ut

b6add46

Signed-off-by: sunby <[email protected]>

sunby force-pushed the add_json_index branch from 43e288f to b6add46 Compare November 7, 2024 06:46

feat: support json index #36750

Are you sure you want to change the base?

feat: support json index #36750

Conversation

sunby commented Oct 10, 2024 • edited Loading

sre-ci-robot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 10, 2024

mergify bot commented Oct 12, 2024

tedxu Oct 14, 2024

Choose a reason for hiding this comment

tedxu Oct 14, 2024

Choose a reason for hiding this comment

tedxu Oct 14, 2024

Choose a reason for hiding this comment

sunby Oct 31, 2024

Choose a reason for hiding this comment

tedxu Oct 14, 2024

Choose a reason for hiding this comment

sunby Oct 31, 2024

Choose a reason for hiding this comment

mergify bot commented Oct 31, 2024

mergify bot commented Oct 31, 2024

codecov bot commented Oct 31, 2024 • edited Loading

Codecov Report

mergify bot commented Oct 31, 2024

xiaofan-luan commented Nov 1, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 4, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 5, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 6, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

sunby commented Oct 10, 2024 •

edited

Loading

codecov bot commented Oct 31, 2024 •

edited

Loading